当前位置： 首页 > news >正文

批量做单页网站推广策划方案怎么写

news 2025/8/8 14:20:56
批量做单页网站,推广策划方案怎么写,前端开发人员怎么做网站,在公司做网站是什么职位文章目录 其他人的内容，笔记写的更好，思路可以去看他们的MapReduceworkermapreduce coordinatorrpc纠错 源码worker.gocoordinator.gorpc.go 原本有可借鉴的部分 mrsequential.go，多看几遍源码 其他人的内容，笔记写的更好&#xf…

文章目录

  • 其他人的内容，笔记写的更好，思路可以去看他们的
  • MapReduce
    • worker
      • map
      • reduce
    • coordinator
    • rpc
    • 纠错
  • 源码
    • worker.go
    • coordinator.go
    • rpc.go

原本有可借鉴的部分
mrsequential.go，多看几遍源码

其他人的内容，笔记写的更好，思路可以去看他们的

MIT - 6.824 全课程 + Lab 博客总览-CSDN博客

MapReuce 详解与复现, 完成 MIT 6.824(6.5840) Lab1 - 掘金 (juejin.cn)

mit 6.824 lab1 笔记 - 掘金 (juejin.cn)

MapReduce

每个 worker 进程需完成以下工作：
向 master 进程请求 task，从若干文件中读取输入数据，执行 task，并将 task 的输出写入到若干文件中。

master 进程除了为 worker 进程分配 task 外，还需要检查在一定时间内（本实验中为 10 秒）每个 worker 进程是否完成了相应的 task，如果未完成的话则将该 task 转交给其他 worker 进程。

worker

worker函数包含map和redicef两个功能
需要知道自己执行哪个功能

worker 请求获取任务 GetTask

任务设置为结构体，其中一个字段为任务类型

type Task struct{Type：0 1 }0为map，1为reduce const枚举

一个为编号

如何获取任务？

GetTask请求coordinator的assignTask方法，传入为自己的信息（？？？），获得任务信息

自己当前的状态 空闲 忙碌

call（rpcname,args,reply）bool的rpcname对应coordinator.go中coordinator的相关方法

???没理解这个什么用
GetTask中调用call来从coordinator获取任务信息？

那么rpc.go来干什么

使用ihash(key) % NReduce为Map发出的每个KeyValue选择reduce任务号。随机选择序号
n个文件，生成m个不同文件 n X m个 文件？？

保存多少个文件

map得到

对结果ohashkey，写入到文件序号1-10

根据序号分配reduce任务

将结果写入到同一文件

map

mapf func(filename string, content string) []KeyValue

filename是传入的文件名
content为传入的文件的内容——传入前需读取内容

传出intermediate[] 产生文件名 mr-x-y

reduce

reducef func(key string, values []string) string)

这里的key对应ihash生成的任务号？？

coordinator

分配任务
需要创建几个map worker—根据几个文件
几个 reduce worker—根据设置，这里为10

coordinator函数用来生成唯一id

Coordinator 结构体定义

用来在不同rpc之间通信，所以其内容是公用的？

type Coordinator struct{files   []stringnReduce int
当前处在什么阶段？ state}type dic struct{
status 0or1or2
id       
}

map[file string]

如何解决并发问题？？

怎么查询worker的状态？？

worker主动向coordinator发送信息

rpc

args 请求参数怎么定义

type args struct{

自己的身份信息

自己的状态信息

}

纠错

  1. 6.5840/mr.writeKVs({0xc000e90000, 0x1462a, 0x16800?}, 0xc00007d100, {0xc39d00, 0x0, 0x0?})
    /home/wang2/6.5840/src/mr/worker.go:109 +0x285

  2. *** Starting wc test.
    panic: runtime error: index out of range [1141634764] with length 0

    ihash取余

  3. runtime error: integer divide by zero

reply.NReduce = c.nReduce // 设置 NReduce

  1. cat: ‘mr-out*’: No such file or directory
    — saw 0 workers rather than 2
    — map parallelism test: FAIL
    cat: ‘mr-out*’: No such file or directory
    — map workers did not run in parallel
    — map parallelism test: FAIL

    cat: ‘mr-out*’: No such file or directory
    — too few parallel reduces.
    — reduce parallelism test: FAIL
    文件句柄问题

  2. sort: cannot read: ‘mr-out*’: No such file or directory
    cmp: EOF on mr-wc-all which is empty
    2024/07/19 11:15:10 dialing:dial unix /var/tmp/5840-mr-1000: connect: connection refused
    2024/07/19 11:15:10 dialing:dial unix /var/tmp/5840-mr-1000: connect: connection refused
    2024/07/19 11:15:10 dialing:dial unix /var/tmp/5840-mr-1000: connect: connection refused

  3. coordinator结构体的并发读取问题
    dialing:dial-http unix /var/tmp/5840-mr-1000: read unix @->/var/tmp/5840-mr-1000: read: connection reset by peer

源码

由于笔记等提示跟没有没区别，这里将源码放上，等大家实在没办法再看吧，
github仓库地址
注意lab1中的提示很重要

博主初期也迷茫过，检查bug也痛苦过，祝福大家。
在这里插入图片描述

有的时候也会出错，但是不想搞了–

在这里插入图片描述

worker.go

package mrimport ("encoding/json""fmt""io""os""sort""strings""time"
)
import "log"
import "net/rpc"
import "hash/fnv"// for sorting by key.
type ByKey []KeyValue// for sorting by key.
func (a ByKey) Len() int           { return len(a) }
func (a ByKey) Swap(i, j int)      { a[i], a[j] = a[j], a[i] }
func (a ByKey) Less(i, j int) bool { return a[i].Key < a[j].Key }const (Map = iotaReduceOver
)
const (Idle = iotaBusyFinish
)// Map functions return a slice of KeyValue.type KeyValue struct {Key   stringValue string
}// use ihash(key) % NReduce to choose the reduce
// task number for each KeyValue emitted by Map.func ihash(key string) int {h := fnv.New32a()h.Write([]byte(key))return int(h.Sum32() & 0x7fffffff)
}// main/mrworker.go calls this function.func Worker(mapf func(string, string) []KeyValue, reducef func(string, []string) string) {// Your worker implementation here.Ch := make(chan bool)for {var Res = &Args{State: Idle} //初始化为idle状态var TaskInformation = &TaskInfo{}GetTask(Res, TaskInformation)//主任务结束后不再请求if TaskInformation.TaskType == Over {break}//fmt.Println("do it!")go DoTask(TaskInformation, mapf, reducef, Ch)sign := <-Ch//fmt.Println("sign:", sign)if sign == true {//fmt.Println("Finish one,ID:", TaskInformation.TaskId)Done(Res, TaskInformation)} else {//TaskInformation.Status = Idle//fmt.Println("err one,ID:", TaskInformation.TaskId)call("Coordinator.Err", Res, TaskInformation)Res = &Args{State: Idle}}time.Sleep(time.Second)}// uncomment to send the Example RPC to the coordinator.//CallExample()}func GetTask(Args *Args, TaskInformation *TaskInfo) {// 调用coordinator获取任务for {call("Coordinator.AssignTask", Args, TaskInformation)//fmt.Println(TaskInformation)if TaskInformation.Status != Idle {Args.State = BusyArgs.Tasktype = TaskInformation.TaskTypeArgs.TaskId = TaskInformation.TaskId//fmt.Println("TaskInfo:", TaskInformation)//fmt.Println("Args：", Args)call("Coordinator.Verify", Args, TaskInformation)break}time.Sleep(time.Second)}//fmt.Printf("Type:%v,Id:%v\n", TaskInformation.TaskType, TaskInformation.TaskId)
}func writeKVs(KVs []KeyValue, info *TaskInfo, fConts []*os.File) {//fConts := make([]io.Writer, info.NReduce)KVset := make([][]KeyValue, info.NReduce)//fmt.Println("start write")//for j := 1; j <= info.NReduce; j++ {////	fileName := fmt.Sprintf("mr-%v-%v", info.TaskId, j)//	os.Create(fileName)////	f, _ := os.Open(fileName)//	fConts[j-1] = f////	defer f.Close()//}var Order intfor _, v := range KVs {Order = ihash(v.Key) % info.NReduceKVset[Order] = append(KVset[Order], v)}//fmt.Println("kvset:", KVset)for i, v := range KVset {for _, value := range v {data, _ := json.Marshal(value)_, err := fConts[i].Write(data)//fmt.Println("data: ", data)//fmt.Println("numbers:", write)if err != nil {return}}}//fmt.Println("finish write")
}func read(filename string) []byte {//fmt.Println("read", filename)file, err := os.Open(filename)defer file.Close()if err != nil {log.Fatalf("cannot open %v", filename)fmt.Println(err)}content, err := io.ReadAll(file)if err != nil {log.Fatalf("cannot read %v", filename)}return content
}// DoTask 执行mapf或者reducef任务func DoTask(info *TaskInfo, mapf func(string, string) []KeyValue, reducef func(string, []string) string, Ch chan bool) {//fConts := make([]io.Writer, info.NReduce)//fmt.Println("start", info.TaskId)//go AssignAnother(Ch)switch info.TaskType {case Map:info.FileContent = string(read(info.FileName))//fmt.Println(info.FileContent)KVs := mapf(info.FileName, info.FileContent.(string))//fmt.Println("map:", KVs)//将其排序sort.Sort(ByKey(KVs))var fConts []*os.File // 修改为 *os.File 类型//0-9for j := 0; j < info.NReduce; j++ {//暂时名，完成后重命名fileName := fmt.Sprintf("mr-%v-%v-test", info.TaskId, j)//_, err := os.Create(fileName)//if err != nil {//	fmt.Println(err)//	return//}////f, _ := os.Open(fileName)//fConts[j-1] = f////defer f.Close()f, err := os.Create(fileName) // 直接使用 Create 函数if err != nil {fmt.Println(err)return}//fmt.Println("creatfile:  ", fileName)//fConts[j] = ffConts = append(fConts, f)defer os.Rename(fileName, strings.TrimSuffix(fileName, "-test"))defer f.Close()}writeKVs(KVs, info, fConts)case Reduce:fileName := fmt.Sprintf("testmr-out-%v", info.TaskId)fileOS, err := os.Create(fileName)//fmt.Println("create success")if err != nil {fmt.Println("Error creating file:", err)return}defer os.Rename(fileName, strings.TrimPrefix(fileName, "test"))defer fileOS.Close()var KVs []KeyValue//读取文件for i := 0; i < info.Nmap; i++ {fileName := fmt.Sprintf("mr-%v-%v", i, info.TaskId)//fmt.Println(fileName)file, err := os.Open(fileName)defer file.Close()if err != nil {fmt.Println(err)}dec := json.NewDecoder(file)for {var kv KeyValueif err := dec.Decode(&kv); err != nil {break}//fmt.Println(kv)KVs = append(KVs, kv)}}//var KVsRes []KeyValuesort.Sort(ByKey(KVs))//整理并传输内容给reducei := 0for i < len(KVs) {j := i + 1for j < len(KVs) && KVs[j].Key == KVs[i].Key {j++}values := []string{}for k := i; k < j; k++ {values = append(values, KVs[k].Value)}// this is the correct format for each line of Reduce output.output := reducef(KVs[i].Key, values)//每个key对应的计数//KVsRes = append(KVsRes, KeyValue{KVs[i].Key, output})fmt.Fprintf(fileOS, "%v %v\n", KVs[i].Key, output)i = j}}Ch <- true}func Done(Arg *Args, Info *TaskInfo) {//Info.Status = Idlecall("Coordinator.WorkerDone", Arg, Info)//arg重新清空Arg = &Args{State: Idle}
}func AssignAnother(Ch chan bool) {time.Sleep(2 * time.Second)Ch <- false
}// example function to show how to make an RPC call to the coordinator.
//
// the RPC argument and reply types are defined in rpc.go.
func CallExample() {// declare an argument structure.args := ExampleArgs{}// fill in the argument(s).args.X = 99// declare a reply structure.reply := ExampleReply{}// send the RPC request, wait for the reply.// the "Coordinator.Example" tells the// receiving server that we'd like to call// the Example() method of struct Coordinator.ok := call("Coordinator.Example", &args, &reply)if ok {// reply.Y should be 100.fmt.Printf("reply.Y %v\n", reply.Y)} else {fmt.Printf("call failed!\n")}
}// send an RPC request to the coordinator, wait for the response.
// usually returns true.
// returns false if something goes wrong.func call(rpcname string, args interface{}, reply interface{}) bool {// c, err := rpc.DialHTTP("tcp", "127.0.0.1"+":1234")sockname := coordinatorSock()//fmt.Println("Worker is dialing", sockname)c, err := rpc.DialHTTP("unix", sockname)if err != nil {//log.Fatal("dialing:", err)return false}defer c.Close()err = c.Call(rpcname, args, reply)if err != nil {//fmt.Println(err)return false}return true
}

coordinator.go

package mrimport ("log""sync""time"
)
import "net"
import "os"
import "net/rpc"
import "net/http"type Coordinator struct {// Your definitions here.files      []stringnReduce    intMapTask    map[int]*TaskReduceTask []intOK         boolLock       sync.Mutex
}type Task struct {fileName stringstate    int
}var TaskMapR map[int]*Taskfunc (c *Coordinator) Verify(Arg *Args, Reply *TaskInfo) error {switch Arg.Tasktype {case Map:time.Sleep(3 * time.Second)if c.MapTask[Arg.TaskId].state != Finish {c.MapTask[Arg.TaskId].state = IdleReply = &TaskInfo{}}case Reduce:time.Sleep(3 * time.Second)if c.ReduceTask[Arg.TaskId] != Finish {c.ReduceTask[Arg.TaskId] = IdleReply = &TaskInfo{}}}return nil
}// Your code here -- RPC handlers for the worker to call.
func (c *Coordinator) AssignTask(Arg *Args, Reply *TaskInfo) error {c.Lock.Lock()defer c.Lock.Unlock()//如果请求为空闲if Arg.State == Idle {//Args.State = Busy//首先分配Mapfor i, task := range c.MapTask {//fmt.Println(*task, "Id:", i)if task.state == Idle {//Arg.Tasktype = Map//Arg.TaskId = i + 1Reply.TaskType = MapReply.FileName = task.fileName//fmt.Println(task.fileName)Reply.TaskId = i          //range从0开始Reply.NReduce = c.nReduce // 设置 NReduceReply.Status = Busytask.state = Busy//fmt.Println("map,Id:", i)return nil}}//Map完成后再Reducefor _, task := range c.MapTask {if task.state != Finish {//fmt.Println("等待Map完成")return nil}}//fmt.Println("MapDone")//分配Reducefor i, v := range c.ReduceTask {//fmt.Println(c.ReduceTask)if v == Idle {Arg.Tasktype = ReduceArg.TaskId = iReply.TaskType = ReduceReply.TaskId = iReply.NReduce = c.nReduce // 设置 NReduceReply.Status = BusyReply.Nmap = len(c.files)c.ReduceTask[i] = Busy//fmt.Println(c.ReduceTask[i])//fmt.Println("reduce", i)return nil}}//Reduce都结束则成功for _, v := range c.ReduceTask {if v == Finish {} else {return nil}}Reply.TaskType = Overc.OK = true}return nil
}func (c *Coordinator) WorkerDone(args *Args, reply *TaskInfo) error {//c.Lock.Lock()//defer c.Lock.Unlock()//reply清空reply = &TaskInfo{}//args.State = Finishid := args.TaskId//fmt.Println("id", id)switch args.Tasktype {case Map:c.MapTask[id].state = Finish//fmt.Println(*c.MapTask[id])case Reduce:c.ReduceTask[id] = Finish//fmt.Println(c.ReduceTask)}return nil
}func (c *Coordinator) Err(args *Args, reply *TaskInfo) error {//c.Lock.Lock()//defer c.Lock.Unlock()reply = &TaskInfo{}id := args.TaskIdswitch args.Tasktype {case Map:if c.MapTask[id].state != Finish {c.MapTask[id].state = Idle}case Reduce:if c.ReduceTask[id] != Finish {c.ReduceTask[id] = Idle}}return nil
}// an example RPC handler.
//
// the RPC argument and reply types are defined in rpc.go.func (c *Coordinator) Example(args *ExampleArgs, reply *ExampleReply) error {reply.Y = args.X + 1return nil
}// start a thread that listens for RPCs from worker.gofunc (c *Coordinator) server() {rpc.Register(c)rpc.HandleHTTP()sockname := coordinatorSock()os.Remove(sockname)l, e := net.Listen("unix", sockname)if e != nil {log.Fatal("listen error:", e)}//fmt.Println("Coordinator is listening on", sockname)go http.Serve(l, nil)
}// main/mrcoordinator.go calls Done() periodically to find out
// if the entire job has finished.func (c *Coordinator) Done() bool {//c.Lock.Lock()//defer c.Lock.Unlock()ret := falseif c.OK == true {ret = true}// Your code here.return ret
}// create a Coordinator.
// main/mrcoordinator.go calls this function.
// nReduce is the number of reduce tasks to use.
func MakeCoordinator(files []string, nReduce int) *Coordinator {//fmt.Println(files)TaskMapR = make(map[int]*Task, len(files))for i, file := range files {TaskMapR[i] = &Task{fileName: file,state:    Idle,}}ReduceMap := make([]int, nReduce)c := Coordinator{files:      files,nReduce:    nReduce,MapTask:    TaskMapR,ReduceTask: ReduceMap,OK:         false,}// Your code here.c.server()return &c
}

rpc.go

package mr//
// RPC definitions.
//
// remember to capitalize all names.
//import "os"
import "strconv"//
// example to show how to declare the arguments
// and reply for an RPC.
//type ExampleArgs struct {X int
}type ExampleReply struct {Y int
}// Add your RPC definitions here.type Args struct {State    intTasktype intTaskId   int
}type TaskInfo struct {Status intTaskType int //任务基本信息TaskId   intNReduce intNmap    intFileName    stringFileContent any//Key    string //reduce所需信息//Values []string
}// Cook up a unique-ish UNIX-domain socket name
// in /var/tmp, for the coordinator.
// Can't use the current directory since
// Athena AFS doesn't support UNIX-domain sockets.func coordinatorSock() string {s := "/var/tmp/5840-mr-"s += strconv.Itoa(os.Getuid())return s
}
http://www.jinmujx.cn/news/108787.html

相关文章：

  • 大型建站网站seo矩阵培训
  • 购买设备有什么网站做参考整合营销案例举例说明
  • 做宣传单页的网站爱站站长工具
  • 网站开发运行环境品牌营销策划网站
  • 做fitting的网站百度系app有哪些
  • 产品展示网站 模板做一个官网要多少钱
  • 做网站总结作文地推拉新app推广平台
  • asp学校网站系统热搜在哪里可以看
  • 网站关键词在哪里做2023年4 5月份疫情结束吗
  • 深圳好的网站建设公司排名网站建设需要啥
  • 如何创建百度网站seo排名技术软件
  • wordpress响应式cms百度竞价推广关键词优化
  • 营销型网站建设开发百度seo词条优化
  • 深圳网站制作 论坛全国十大教育机构
  • 广西企业网站有哪些百度app下载安装官方免费下载
  • 东莞市官网网站建设腾讯朋友圈广告投放价格
  • 电商行业河源seo
  • 任丘网站开发建设怎么选武汉seo学徒
  • 山西营销型企业网站开发网站域名ip查询
  • 源码下载网站推荐googleplay
  • 做英文网站哪家好输入关键词就能写文章的软件
  • 南开大学 网站开发技术 刘冲淘宝关键词指数
  • 高大上强企业网站甘肃网站推广
  • 建设网站的五个步骤seo是干什么的
  • 做seo需要建网站吗电商seo是什么意思啊
  • 临朐网站建设定制网络推广服务商
  • 站长工具seo下载免费发布信息网网站
  • 聊城市建设学校福州关键词优化平台
  • 企业名录联系方式查询平台seo外包公司兴田德润官方地址
  • 在什么网站可以做外贸出口劳保鞋微信crm系统软件